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(57) ABSTRACT 

In a packet-switched network, a distributed performance 
measurement system is disclosed. In each node, a node 
monitor maintains information for each virtual channel 
passing through the node, including a list of the arrival times 
of the packets queued for the virtual channel. In addition, 
each node determines node performance characteristics, 
which may include throughput, effective rate, delay, loss 
rate, and occupancy. The network further includes a network 
monitor, which periodically collects the node performance 
characteristics from each node and determines performance 
characteristics for each virtual channel in the network, 
end-to-end. The network performance characteristics may 
include throughput, effective rate, delay, loss rate, and 
occupancy. The results are useful for network operators and 
users to optimize the network and/or their use of the net- 
work. 
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METHOD AND APPARATUS FOR 
MEASURING PERFORMANCE IN PACKET- 
SWITCHED NETWORKS 



FIELD OF INVENTION 

The invention relates generally to computer and commu- 
nication networks. More specifically, the invention relates to 
measuring the performance of such networks. 

BACKGROUND 

In communications technology, there is an ever- 
increasing demand for high-performance networks, and in 
particular, a demand for high-performance Internet access. 
This increased demand has led to the development of 
improved networks capable of handling larger volumes of 
data with smaller delays. 

Communications networks, such as ATM networks or the 
Internet, are generally formed with a number of transmission 
links interconnected with switches. A transmission link is 
any medium through which signals are communicated and 
can be single or multiple twisted pairs, optical fiber, coaxial 
cable, radio links, or other mediums. A switch is a device 
with one or more input ports and one or more output ports. 
The switch directs bits arriving at an input port to the 
appropriate output port. Switching in communications is 
accomplished using one of two methods: circuit switching 
and packet switching. Only packet switching is relevant for 
the purposes of the present discussion. 

In packet switching, the data in a data stream is divided 
into "packets." In addition to a portion of data, each packet 
carries various identifying information about the data. Such 
identifying information includes routing information, for 
instance, source and destination addresses for the data. 
When a packet arrives at a switch, it is stored in a buffer. The 
switch looks at the routing information in the packet and 
routes the packet to the appropriate output port. The packets 
are then forwarded one "hop" (or switch) at a time, until they 
reach their destination. 

Generally in packet switching, each packet in a particular 
data stream does not necessarily follow the same path to its 
destination as the others. Nonetheless, certain types of 
packet switching, e.g., ATM, do specify the path that the 
packets are to follow between the switches for a particular 
end-to-end connection. Such a predetermined path is typi- 
cally referred to as a "virtual channel." 

Unlike general packet switching where packets are not 
guaranteed to arrive in the order they were sent, when virtual 
channels are specified, all of the packets follow the same 
route along the virtual channel, thereby guaranteeing the 
order of arrival. These "virtual channels" have greatly 
enhanced the performance of networks in using resources 
efficiently and can usually provide a minimum "quality of 
service", i.e., a minimum bandwidth provided between two 
points, A and B. 

In order to maintain this quality of service, switches 
monitor packets that pass through them. For instance, 
switches will typically count the packets they transport per 
port and/or virtual channel and count the number of packets 
that are dropped. This monitoring allows a particular switch 
to send messages to other switches. For instance, if a switch 
becomes overloaded or saturated with packets it may notify 
other upstream switches to slow down or stop transmission. 
Or this monitoring may cause a switch to take other actions. 
For instance, if too many packets arrive for a particular 
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virtual channel, the switch may drop some of them in order 
to keep the virtual channel from usurping the transmission 
links from other virtual channels (and thereby deny the other 
virtual channels their minimum quality of service). 

5 Yet, while this monitoring is sufficient to ensure a mini- 
mum quality of service for each virtual channel, this moni- 
toring generally does not evaluate the actual performance or 
usage of a switch, a virtual channel, or the network as a 
whole. Hence, a network operator cannot readily determine 

io what resources are available on a network, what resources 
are being wasted, or if a switch is not performing optimally. 
The provision of such information has generally been 
deemed too costly and complicated. 

35 SUMMARY 

A system and method in accordance with the invention 
provide a scheme by which the performance of a network 
can be easily monitored. In general, an embodiment of the 
invention determines values related to the length of time 

20 packets are present in a node, including effective rale for a 
virtual channel in a node and/or delay for a virtual channel 
in a node. "Effective rate" is used herein to mean the 
throughput rate during the time when a given virtual channel 
is busy, having packets in its queue. 

25 More specifically, some embodiments of the invention are 
used in a packet-switched network comprised of a plurality 
of nodes. In each node there is a node monitor. In one 
embodiment, the node monitor maintains information for 
each virtual channel that passes through the node. The 

30 information maintained includes the time of arrival of pack- 
ets in the queue for the virtual channel. In one embodiment, 
such information is maintained in a linked list. The node 
monitor further utilizes the information maintained to deter- 
mine node performance characteristics. In one embodiment, 

35 these node performance characteristics include throughput, 
effective rate, loss rate, delay, and occupancy for each virtual 
channel carried by the node. 

In some embodiments, the node monitor is split into an 

40 epoch monitor and an aggregate monitor. The epoch monitor 
monitors the packets as described above and performs 
simple calculations such as counting, adding, and compar- 
ing. The aggregate monitor performs more processing- 
intensive calculations. 

45 A network in accordance with one embodiment of the 
invention further includes a network monitor. Periodically, 
the network monitor receives the node performance charac- 
teristics from each node and then determines overall per- 
formance characteristics for the virtual channels in the 

50 network from end to end (from the virtual channel source to 
its destination). This information may then be provided in a 
report to a network operator or one or more users, who can 
then use the information to optimize the network and/or their 
use of the network. 

55 Hence, in accordance with one embodiment of the 
invention, system performance measurements are taken in a 
distributed manner, e.g., low-level (in the epoch monitor), 
mid-level (in the aggregate monitor), and network (in the 
network monitor). Such a distributed performance measure - 

60 ment system allows network performance to be monitored 
and avoids interference with packet flow without taxing the 
resources of any one node or the network overall. 

BRIEF DESCRIPTION OF THE DRAWINGS 

65 The present invention is described with respect to par- 
ticular exemplary embodiments thereof and reference is 
accordingly made to the drawings in which: 
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FIG. 1 is a generalized block diagram of a network in 
accordance with one embodiment of the invention; 

FIG. 2 is a generalized block diagram of a switch in 
accordance with one embodiment of the invention; 

FIG. 3 is a generalized block diagram of a node monitor 
in accordance with one embodiment of the invention; 

FIG. 4 is a representational diagram of the data structures 
utilized in accordance with one embodiment of the invention 
to monitor packets; 

FIG. 5 is an illustration of the determination of busy time 
in accordance with one embodiment of the invention; and 

FIG. 6 is a diagram utilized to illustrate a delay fluctuation 
calculation in accordance with one embodiment of the 
invention. 

DETAILED DESCRIPTION 

In order to overcome the disadvantages discussed above, 
a method and system in accordance with the invention 
provides a simple and relatively low cost way to monitor a 
network, its performance and its available resources. In 
general, an embodiment of the invention maintains moni- 
toring circuitry in every switch or other node to monitor all 
packets that flow through the node. As well, an embodiment 
in accordance with the invention maintains three levels of 
calculations and measurements: one low level set performed 
at the node, a second more calculation intensive set also 
performed at the node, and a high level set for evaluating 
characteristics of the entire network. This distributed mea- 
surement system allows resources to be allocated efficiently 
and-avoids-interference-with-normal-packet.flow through.the 
switch. Details are fully described below. 

FIG. 1 shows a network in accordance with the invention. 
Network 100 is composed of a plurality of switches and 
other nodes 102, Z1-Z5, generally referred to herein as 
"nodes." Each node is connected to one or more other nodes 
by way of a transmission links 104. In addition, a network 
monitor node (NMN) 106 is in communication with one or 
more nodes 102. 

The nodes 102 are utilized in virtual channels. As 
described previously, a virtual channel is a predesignated 
path for packets to follow from a source to a destination. For 
example, referring to FIG. 1, packets transmitted from node 
Zl to node Z4 may follow a path (a virtual channel) defined 
by the transmission links from Zl to Z5 and then from Z5 
to Z4. Alternatively, a virtual channel from Zl to Z4 could 
be set up using the link from Zl to Z2, then the link from Z2 
to Z3, then the link from Z3 to Z4. In either case, for the 
particular connection from Zl to Z4, packets will always 
follow the same path. Virtual channels and the processes for 
their establishment are well known in the art and are often 
used in communications protocols such as ATM 
(asynchronous transfer mode) or IP (Internet protocol), 
which uses MPLS (multiprotocol label switching). 

Each node 102 in network 100 includes hardware, 
software, and/or firmware for monitoring packets as they 
pass through each node and for performing certain analyses 
on the data collected by monitoring. After a specified period 
of time, the network monitoring node 106, also including 
hardware, software, and/or firmware, collects information 
from the nodes 102 for further analysis and report prepara- 
tion. These reports are useful to allow a network operator or 
user to evaluate network performance and available 
resources. Further details of the monitoring and analysis 
process are discussed below. 

A generalized block diagram of a node 102 in accordance 
with an embodiment of the invention is shown in FIG. 2. As 
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shown, each node has one or more input ports 202 and one 
or more output ports 204. Four input ports 202 and four 
output ports 204 are shown for the purposes of illustration. 
In addition, each node includes switching logic 205, which 
includes all of the circuitry, memory, databases, instructions, 
and any other requirements for routing a packet from an 
input port 202 to the appropriate output port 204 in accor- 
dance with the virtual channel along which the packet is to 
be routed. As packets arrive, the switching logic 205 parses 
the packet and determines the virtual channel to which the 
packet belongs. The switching logic 205 then places the 
packet into an appropriate outgoing queue for the particular 
virtual channel. In the event that an output port services 
more than one virtual channel, packets from the queue will 
be multiplexed in some fashion on the outgoing transmission 
link as is understood in the art. 

In addition, in accordance with one embodiment of the 
invention, each node includes a node monitor 206 for 
passively monitoring packets as they pass through node 102. 
More specifically, as shown in FIG. 3, in one embodiment 
each node monitor includes an epoch monitor 302 and an 
aggregate monitor 304. The epoch monitor 302 passively 
monitors the packets as they pass through node 102 and 
maintains an image of each virtual channel queue by using 
a set of linked lists. Two data structures, shown in FIGS. 4a 
and 4b, maintain this set of linked lists: one linked list per 
each virtual channel passing through the node plus one 
linked list of "free" locations. The linked list for a particular 
virtual channel specifies the arrival times of the packets that 
are currently in the node for that virtual channel in the order 
of arrival. 

In particular, for each virtual channel the data structure of 
FIG. 4a maintains an entry pointing to an address in the data 
structure of FIG. 4b that is representative of the first-to- 
arrive packet in the queue of the virtual channel, a "head" 
packet. As well, the data structure of FIG. 4a maintains an 
entry pointing to an address in the data structure of FIG. 4b 
that is representative of the last -to -arrive packet in the queue 
for the same virtual channel, a "tail" packet. 

The data structure of FIG, 4b maintains linked lists of 
information corresponding to each virtual channel queue. In 
particular, in one embodiment each packet is represented by 
two entries: a time stamp, indicative of the packet arrival 
time at the node, and a pointer to the address containing 
information about the next packet in the virtual channel's 
queue. Other embodiments may contain additional informa- 
tion for packets. 

For example, the data structure of FIG. 4a shows that for 
virtual channel 1 (where each row of entries in the data 
structure corresponds to a distinct virtual channel) the first 
entry in the data structure of FIG. 4b is located at addr-4 and 
the last entry in the data structure of FIG. 4b is at addr-5. The 
head of the linked list, addr-4, shows an arrival time of time 
tl for the first packet in the queue and points to addr-9, the 
address holding information regarding the second packet in 
the queue. Addr-9 shows an arrival time of time t2 for the 
second packet in the queue and points to addr-1, the address 
holding information regarding the third packet in the queue. 
Addr-1 shows an arrival time of time t3 for the third packet 
and points to addr-11, the address holding information 
regarding the fourth packet in the queue. Addr-11 shows an 
arrival time of time t4 for the fourth packet and points to 
addr-5, the address holding information for the fifth and last 
packet in the queue for virtual channel 1. Addr-5 shows an 
arrival time of time t5 for the fifth packet and also includes 
an end of list indicator, indicating that there are no more 
packets in the queue. 
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In addition to a linked list for each virtual channel, the embodiments of the invention, the number of bytes are 
data structures of FIGS. 4a and 4b also maintain a "free" list, monitored instead of the number of cells. In either case, the 
a linked list of the unused address spaces in the data number of bytes can be easily determined by maintaining a 
structure of FIG. 4b. Like the entries for virtual channels, cell count. In addition, some embodiments of the invention 
FIG. 4a gives an entry for the first free address in the linked s may maintain a cell count by counting packets as they arrive 
list of free addresses (e.g., addr-7) and an entry for the last at the node instead of when they leave, 
free address (e.g., addr-3). Another measurement taken by epoch monitor 302 in one 

At each event (the arrival or departure of a packet from embodiment is the Number of Packets, P„, that leave at the 
the node), the data structures of FIG. 4 are updated. For node 102 during a particular epoch. Like C„, P„ can alter- 
instance, upon the arrival at time t6 of a new packet for 30 natively be determined in some embodiments as packets 
virtual channel 1, the arrival time t6 of the packet is written arrive. 

in the first address location of the free list, e.g., addr-7, in the g US y Time, T„, refers to the time when the queue for the 
data structure of FIG. 4b. The second entry of addr-7 is virtual channel being monitored is not empty. Busy time can 
changed from addr-13 to "end" and the second entry at be computed utilizing the entries in the data structure of FIG. 
location addr-5 is changed from "end" to addr-7. Moreover, 15 4 fl> For instance, referring to FIG. 5, for virtual channel 1, 
in the data structure of FIG. 4a, the head entry of the free list assume that for a particular epoch there are two bursts of 
is changed from addr-7 to addr-13 and the tail entry for data: the first starting at time bl and ending at time b2 and 
virtual channel 1 is changed from addr-5 to addr-7. the second starting at time b3 and ending at time b4. Using 

Although only entries are discussed in FIGS. 4a and 4b the information of FIG. 4a, in one embodiment busy time 
with respect to virtual channel 1 and the free list, it is to be 20 can be determined by subtracting bl from the current time, 
understood that entries for all virtual channels for the node When the queue becomes empty again the busy time value 
will be found in these two data structures in many embodi- is stored, for instance in a register. At time b3, when the 
ments of the invention. Virtual channel 1 entries and free list queue becomes busy again, busy time is determined by 
entries are shown without entries for other virtual channels subtracting b3 from the current time and adding the result to 
for clarity of the illustration. 25 the previous busy time, which has been stored. 

In addition to the head and tail entries corresponding to Peak Delay, D„, refers to the highest delay time a packet 
the linked list representing the queue for each virtual in the queue experiences from the time of its arrival (marked 
channel, the data structure of FIG. 4a further maintains an with a time stamp in FIG. 4b) to the time of its departure, 
entry for each virtual channel entitled "Start of busy cycle." 3Q Delay can be easily determined by subtracting the arrival 
Start of busy -cycle -indicates-the time at -which-the virtual time, stored in-FIG. 4fc, of a-packet from. its departure time, 
channel became busy and the last time that the queue was Peak delay is then determined by taking the maximum of the 
empty. delays determined for all packets in the epoch. 

Using the information stored in the data structures of Total Delay, S M , refers to the sum of the delays from 
FIGS. 4a and 4b, each epoch monitor 302 (FIG. 3) maintains 35 arrival to departure at the node for each packet in the virtual 
measurements for each virtual channel. These measurements channel queue during a particular epoch, 
are listed in Table 1 below: Number of Lost Packets, L n , refers to the number of 

packets dropped or otherwise lost for a particular queue. And 
Maximum Occupancy, Q„, refers to the maximum number 
40 of packets in the queue during a particular epoch. 

It is to be understood that many embodiments of the 
invention do not require maintenance of each value listed in 
Table 1. 

At the epoch monitor, it is desirable to be able to maintain 
these measurements as the packets arrive/depart. Hence, it is 
desirable that the hardware, software, and/or firmware used 
for these measurements be able to perform quickly. 
Therefore, hardware, software, and/or firmware used to 
maintain these measurements is fast, but has little processing 

Each of the above measurements is taken over one or '° P°^.( wh | cb F ca , use f? crifice t0 J"? 0 ? 

more epochs. An "epoch," as used herein, refers to a unit of embodiment of the invention the measurements of Tab e 1 

time, whatever that unit is selected to be. For instance, in one »" li * en and maintained easily by ustng counters registers 

... . c . .. . • n , t (or other memory units), adders, and/or comparators. 

embodiment of the invention, an epoch is one second. In v J 71 r 

another embodiment of the invention, an epoch may be one 55 In one embodiment, after a predesignated period in which 

tenth of a second. An epoch is sometimes referred to herein a number N of epochs (Nx) have passed, the epoch monitor 

in shorthand ast. The subscripted "n" attached toeach of the 302 passes its measurements on to aggregate monitor 304. 

above measurement names refers to an epoch number. For At that time, epoch monitor resets all of its measurements 

instance, for the first monitored epoch, n-1, for the second and begins again. 

monitored epoch, n«2, etc. 60 Aggregate monitor 304, however, further processes the 

Because packets are often a variable length, in some data transferred to it from epoch monitor 302. Unlike epoch 

embodiments of the invention for ease of manipulation monitor 302, aggregate monitor 304 has more powerful 

packets are subdivided into "cells" on arrival at a node. A processing capabilities and may in some embodiments 

cell is 48 bytes in one embodiment of the invention. The include a DSP among other hardware, software, and/or 

cells are reassembled into packets before leaving the node. 65 firmware. Aggregate monitor 304 utilizes the information 

Hence one measurement taken is the Number of Cells, C„, received from epoch monitor 302 to obtain the following 

that leave the node 102 during a particular epoch. In some measurements for the given period as listed in Table 2. 



TABLE 1 


EPOCH MONITOR 




Description 


Name 


Number of Cells 


c n 


Number of Packets 




Busy Time 


T n 


Peak Delay 


D„ 


Total Delays of the P n packets 


s n 


Number of Lost Packets 


K 


Maximum Occupancy of Queue 





45 
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TABLE 2 



Description 



AGGREGATE MONITOR 



Name/Definition 



Total Number of Cells 
Total Number of Packets 
Total Busy Time 
Total Lost Packets 
Average Throughput 
Peak Throughput 
Average Effective Rate 
Minimum Effective Rate 
Maximum Occupancy 
Average Loss Rate 
Average Delay 

Delay Deviation 



Peak Delay 

Average Delays over First M 
Epochs 

Average Delays Over 



C o d + . 

PoP 1 + , 

T = Ti + , 
L = Lj + , 



ATh = C/Nt 
PTh = max {Cj, . . 
AEff - C/T 
MEff - min {Cj/Tl 
Q - max {Qj, . . . , 
LR - L/P 

A - {Si + ... + Sn}/P 



C N }/t 
Q/T N } 



On} 



DD 



N 

■I 

1 n=l 



PD - max {D„ 
A, - (S t + ... ■ 



N-L 



.-,Pn} 
- Sm)/(Pi + 



-Pm) 



A 2 , . . . , A K (defined similarly to Aj 
Consecutive Sets of M Epochs Aq represents the average delay over 
the M epochs that precede the current 
period. 
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The Total Number of Cells, C, for the period is computed 
by taking each C M (where n«l, . . . f N) and summing. The 
Total Number of Packets, P, for the period is computed by 
taking each P„ and summing. The Total Busy Time, T, and 
Total Lost Packets, L, are computed in similar manner. 

"Throughput" is the total number of cells (bytes) to pass 
through the switch in a given time period. In one 
embodiment, aggregate monitor determines Average 
Throughput by dividing the Total Number of Cells C by Nx. 
The Peak Throughput is determined by dividing the maxi- 
mum number of cells (bytes) that leave the switch during 
any epoch of the period by x. 

Effective Rate is defined herein as the throughput rate 
when there was actually data in the queue. In one 
embodiment, the Average Effective Rate is determined by 
dividing the Total Number of Cells C by the Total Busy Time 
T The Minimum Effective Rate is calculated by determining 
C n fT„ for each epoch and then taking the minimum. 

Maximum Occupancy Q is determined by taking the 
maximum of the Maximum Occupancies Q„ for all epochs 
in the current period. And the Average Loss Rate is deter- 
mined by dividing the Total Lost Packets L by the Total 
Number of Packets P. 

The Average Delay is determined by summing the Total 
Delays S„ for each epoch and dividing by the Number of 
Packets P in the current period. Delay Deviation is the 
standard deviation of the average delays per epoch over N 
epochs. Peak Delay is determined by taking the maximum of 
the Peak Delays for all epochs in the current period. 

Finally, in one embodiment the aggregate monitor per- 
forms an additional set of unique calculations relating to 
Delay Fluctuation. Delay Fluctuation is calculated to deter- 
mine how good the data related to delay is for a given time 
period. In other words, if the delay values fluctuate 
considerably, the delay information for that time period may 
not be as relevant as that in other time periods. 
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One way to estimate the variability of the delay is to 
calculate the delay deviation. To do that calculation, 
however, requires the storage of all delay values S 3 , . . . , 
for Nx seconds before the average A can be found and before 
the calculation can be performed. 

Therefore, in one embodiment of the invention a method 
for calculating Delay Fluctuation is utilized that does not 
require such storage. Instead, for every epoch, the Delay 
Fluctuation is calculated by taking the absolute value of the 
difference between the average delay over that epoch (S„) 
and the average delay over the previous M epochs, where 
N-KM and K is rather large (e.g., K-10). FIG. 6 illustrates, 
showing a graph where the X-axis represents time in epochs 
and the Y-axis represents delay. As shown, a period defined 
by Nt is divided into K sets of Mx. First, the average delay 
for each epoch in a set of M epochs (for instance, the fourth 
set) is compared to the average delay over the previous set 
of M epochs (e.g., the third set of M epochs) by subtracting 
the two and taking the absolute value. This comparison is 
performed for each epoch in the current set of M epochs 
(e.g., the fourth set) and the results are then summed. This 
process is repeated for all sets of M epochs in the time period 
and the results from each set of M epochs are summed, 
where M=l . . . K. These comparisons and sums can be 
expressed as follows: 



ZZ 



is-the specific 



and where Aj_ x is the 



wherc-j is thc-jth set of M epochs^ where 
epoch in the jth set of M epochs, 

average over the previous M epochs, and is calculated as 
described above by summing the delays for each epoch and 
dividing by the total number of packets in the set of M 
epochs. The average delay for M epochs is as follows: 



Z Pmu-w 



Although a number of measurements and/or calculations 
are described above, it is to be understood that not all of 
these measurements and/or calculations need be performed 
by all embodiments of the invention. 

To determine how each virtual channel and the network is 
performing overall, the network monitor 106 periodically 
polls the nodes 102 to fetch the data stored by aggregate 
monitor 304. In one embodiment data is stored in the node 
utilizing a double buffering scheme: node 102 first writes a 
first buffer and, when done (either when full or when a first 
set of data is complete), will write to a second buffer. When 
the node 102 writes to the second buffer, the network 
monitor 106 is free to retrieve data from the first buffer, and 
vice versa. Other embodiments of the invention may use 
other methods to provide "slack** time to allow for retrieval 
of data by the network monitor 106. 

In one embodiment, the data retrieved or otherwise 
received from node 102 is provided with a version number 
by the node to prevent confusion to the network monitor 
106. The network monitor then combines the information it 
receives from all nodes, computing the following quantities 
for each entire virtual channel, end-to-end, shown in Table 
3. It is to be understood that not all embodiments will require 
the computation of each quantity lusted in Table 3. It is to be 
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understood that not all embodiments will require the com- 
putation of each quantity listed in Table 3. 

TABLE 3 

Virtual Channel 



Throughput 


Average (ATh vc ) 


Should be approximately the 






same at all nodes; determined 






using the Average Throughout 






determined by the first node in 






the virtual channel 






ATh vc m Ath(z), where z - 1 




Peak (PTtO 


maximum of Peak 






Throughputs determined by 






each node 






PTtivc = max {Th(z)} 






where z = 1, . . . , Z 


Effective Rate 


Average (AEff vc ) 


minimum of Average Effective 






Rates determined by each 






node 






AEff vc - min {AEff(z)}, 






where z - 1, . . . , Z 




Minimum (MEff V0 ) 


minimum of Minimum 




Effective Rate determined by 






each node 






Meff vc * min \wir.a\p.)j 






where z = 1, . . . , Z 


Delay 


Average (Ayj 


sum of Average Delay 




determined by each node 






z 

2=1 




Deviation Fluctuation 


The sum of the Fluctuations 






determined at- each-node for 




the virtual channel 






z 

DF^ = Yj df ( z )> 
i=i 




Peak (PD VC ) 


Average Delay for the virtual 






channel plus three times 






Fluctuation for the virtual 






channel 






PD VC o Av C + 3DF VC 


Loss Rate 


Average (LR^ 


Sum of the Average Loss 






Rates determined by each 






node. 






z 






LRvc = ^ LR(z), 
i=i 


Occupancy 


Maximum per node 


maximum of the Maximum 




(Qvc) 


Occupancies calculated by 






each node. 






Q vc - max {Q(z)}, 






where z - 1, . . . , Z 



10 



25 



40 



45 



50 



At the network, throughput is determined for a given 
virtual channel. Average Throughput should generally be the 
same or similar at each node, so the determination of the 
Average Throughput from any node can be utilized. Table 3 
suggests using the Average Throughput determined by the 
first node in the virtual channel, but the Average Throughput 
determined by any node will be satisfactory. Peak Through- 
put for the virtual channel is determined by comparing the 
Peak Throughput determined by each node and taking the 
maximum. As shown in Table 3 for Peak Throughput and 
other determined values, z is representative of a given node 
in a virtual channel, when the virtual channel includes nodes 
1 through node Z. 

Effective rate is also determined for each virtual channel. 
Average Effective Rate for a virtual channel is determined in 
one embodiment by comparing the Average Effective Rate 



55 



60 



determined by each node and taking the minimum value. 
Minimum Effective Rate for the virtual channel is deter- 
mined by comparing the Minimum Effective Rate deter- 
mined by each node and taking the minimum. 

For each virtual channel, in one embodiment, various 
delay measurements are also determined. Average Delay for 
a virtual channel is determined by summing the Average 
Delay values determined at each node. Delay Fluctuation is 
determined by summing the Delay Fluctuation determined 
by each node. Peak Delay for the virtual channel is then 
conservatively determined by summing the Average Delay 
for the virtual channel with three times the Fluctuation for 
the virtual channel. Peak Delay is thus an estimate and 
assumes a Gaussian distribution. 

Loss rate for the virtual channel is determined by sum- 
ming the Average Loss Rates determined by each node. And 
Occupancy for the virtual channel is determined by com- 
paring the Maximum Occupancy determined at each node 
and taking the maximum. 

The measurements made by the network monitor 106 for 
the virtual channels can be made more accurate by synchro- 
nizing the node monitors 206 in each node 102. Synchro- 
nization ensures that the measurement epochs in the nodes 
correspond to approximately the same packets that flow 
through the nodes. For instance, if node Zl monitors packets 
1, 2, 3, 4, and 5 in a first epoch, then it is desirable that the 
next node in the virtual channel, say node Z2, will monitor 
the same packets 1-5 in a corresponding epoch. One method 
to achieve such synchronization uses the network time 
protocol, which is well known in the art and commonly used 
by computers on the~Internet for clock" synchronization. An 
alternative method for synchronization used in other 
embodiments of the invention is for the network monitor 
106 to periodically, e.g., once a day, send a synchronization 
command to all the node monitors 206. 

In one embodiment, the network monitor node prepares 
reports for use by the network operator and/or other network 
users based upon the data in Table 3, A network operator 
may use the information provided in these reports to verify 
and improve network operation. For instance, the operator 
can adjust the "overbooking" of particular links to satisfy 
quality of service promised to particular users while opti- 
mizing his/her revenues. The operator may also use this 
information to implement usage -based billing. Alternatively, 
if a network user receives this information, the network user 
may be able to adjust his/her virtual channel requests. For 
instance, the user may discover his/her actual needs are less 
than expected. These are only a few of the possible uses the 
information provided may permit. 

Hence a method and system has been described for 
monitoring the performance characteristics of an entire 
network and doing so without placing undue burdens on any 
nodes or influencing the flow of data. Further, a system and 
method in accordance with the invention is relatively low 
cost and easily implemented. 

It should be understood that the particular embodiments 
described above are only illustrative of the principles of the 
present invention, and various modifications could be made 
by those skilled in the art without departing from the scope 
and spirit of the invention. Thus, the scope of the present 
invention is limited only by the claims that follow. 

What is claimed is: 

1. A method comprising: 

determining an effective rate for a virtual channel in a 
node including determining the total number of cells to 
pass through the node during a predetermined period of 
time, and 
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calculating a portion of the predetermined period of time 
during which a queue for the virtual channel of the node 
is not empty the effective rate based on a throughput 
rate when there Is data in a queue corresponding to the 
virtual channel; and 

providing the determined effective rate for a virtual chan- 
nel to a network monitor. 

2. A method for use with a network that transmits packets 
through virtual channels, comprising: 

determining a value related to the length of time one or 
more packets are present in a node wherein determining 
the value further includes determining effective rate for 
a virtual channel in the node and delay for the virtual 
channel in the node, wherein effective rate is based on 
throughput rate when there is data in a queue corre- 
sponding to the virtual channel. 

3. The method of claim 2, wherein determining effective 
rate for a virtual channel in a node includes determining 
busy time for said virtual channel in said node. 

4. The method of claim 2, further including: 
determining effective rate for a virtual channel from end 

to end; 

determining delay for a virtual channel from end to end. 

5. The method of claim 4, wherein delay includes at least 
one of average delay or peak delay. 

6. The method of claim 2, wherein determining delay for 
a virtual channel in a node includes maintaining a linked list 
of information representative of a queue for said virtual 
channel, wherein said information includes the arrival time 
of. each packet. 

7. A system for use with a network for transmitting 
packets trough virtual channels, wherein said network 
includes a plurality of nodes, said system designed to 
measure effective rate for a virtual channel in a node and for 
a virtual channel end-to-end, wherein effective rate is based 
on throughput rate when there is data in a queue correspond- 
ing to the via channel, and designed to measure delay for a 
virtual channel in a node and for a virtual channel end-to- 
end. 

8. The system of claim 7, wherein said system is further 
designed to measure busy time for a node. 

9. The system of claim 7, wherein delay includes at least 
one of average delay or peak delay. 

10. The system of claim 9, wherein delay for a virtual 
channel in a node further includes delay fluctuation. 

11. A system for use with a network for transmitting 
packets through virtual channels, wherein said network 
includes a plurality of nodes, said system comprising: 

means for measuring effective rate for a virtual channel in 
a node, wherein effective rate is based on throughput 
rate when there is data in a queue corresponding to the 
virtual channel; and 

means for measuring delay for a virtual channel in a node. 

12. The system of claim 11, further including: 

means for measuring effective rate for a virtual channel 
end-to-end; and 

means for measuring delay for a virtual channel end-to- 
end. 

13. The system of clam 11, wherein delay includes 
average delay, peak delay, and delay fluctuation. 

14. The system of claim 11, further including: 
means for synchronizing said plurality of nodes. 

15. A node for use in a packet-switched network that 
utilizes virtual channels, including: 

a node monitor designed to maintain information for each 
of a plurality of virtual channels passing through said 
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node, said information including the time of arrival of 
packets in a virtual channel queue in said node, and said 
node monitor designed to utilize said information to 
determine node performance characteristics of each 
5 virtual channel in said node, said node performance 
characteristics including at least one of the group 
including effective rate for a virtual channel in said 
node, wherein effective rate is based on throughput rate 
when there is data in a queue corresponding to the 
virtual channel and delay for said virtual channel in 
said node. 

16. The node of claim 15, wherein said node monitor 
includes: 

an epoch monitor, wherein said epoch monitor monitors 
the arrival and departure of packets to and from said 

15 node over N epochs, maintains information for each of 
said packets, and, using said information, performs 
simple calculations to determine epoch-level perfor- 
mance characteristics over each epoch; 
an aggregate monitor, wherein said aggregate monitor is 

20 in communication with said epoch monitor and, after 
said N epochs, receives said epoch-level performance 
characteristics from said epoch monitor and, using said 
epoch-level performance characteristic, performs more 
processing-intensive calculations than said epoch 

25 monitor to determine node-level performance charac- 
teristics. 

17. The node of claim 15, wherein said node monitor 
includes a linked list of the arrival times of said packets for 
each virtual channel queue in said node. 

30 18. The node of claim 15, wherein delay includes average 
delay and peak delayr 

19, The node of claim 18, wherein said node performance 
characteristics further include throughput, loss rate, and 
occupancy. 

35 20. A packet-switched network including a plurality of 
nodes, comprising: 

a distributed performance measurement system, including 

a) a node monitor in each node including 

i) an epoch monitor, wherein said epoch monitor 
40 monitors the arrival and departure of packets to 

and from said node over N epochs, maintains 
information for each of said packets, and, using 
said information, performs simple calculations to 
determine epoch -level performance characteris- 
45 tics over each epoch; 

ii) an aggregate monitor, wherein said aggregate 
monitor is in communication with said epoch 
monitor and, after said N epochs, receives said 
epoch -level performance characteristics from said 

50 epoch monitor and, using said epoch-level perfor- 

mance characteristics, performs more processing- 
intensive calculations than said epoch monitor to 
determine node-level performance characteristics; 

b) a network monitor, wherein said network monitor is 
55 in communication with each of said plurality of 

nodes and periodically receives said node-level per- 
formance characteristics from each node, and, using 
said node-level performance characteristics, per- 
forms calculations to determine network-level pcr- 
60 formance characteristics. 

21. The network of claim 20, wherein said network 
monitor is designed to initiate synchronization of said nodes. 

22. The network of claim 20, wherein epoch -level per- 
formance characteristics include busy time and delay for a 

65 virtual channel in a node. 

23. The network of claim 22, wherein epoch -level per- 
formance characteristics further include number of cells, 
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number of packets, number of lost packets, and occupancy 
for a virtual channel in a node, and wherein delay includes 
peak delay and total delay for all packets in a given epoch 
for a virtual channel in a node. 

24. The network of claim 20, wherein node-level perforce 
characteristics include effective rate and delay for a virtual 
channel in a node. 

25. The network of claim 24, wherein no level perfor- 
mance characteristics further include throughput, 
occupancy, and loss rate for a virtual channel in a node. 

26. The network of claim 25, wherein node -level perfor- 
mance characteristics further include delay fluctuation. 

27. The network of claim 20, wherein network-level 
performance characteristics include effective rate and delay 
for a virtual channel end-to-end. 

28. The network of claim 27, wherein network-level 
performance characteristics further include throughput, loss 
rate, and occupancy for a virtual channel end-to-end. 

29. The method of claim 1, wherein determining the 
portion of the predetermined period of time during which a 
queue for the viral channel of the node is not empty 
comprises maintaining a data structure representing the 
virtual channel, the data structure listing one or more arrival 
times for one or more cells of the virtual channel. 

30. The method of claim 29, wherein the data structure 
listing one or more arrival times for one or more cells of the 
virtual channel is structured as a linked list. 
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31. An article of manufacture comprising a machine 
accessible medium having content to provide instructions to 
cause an electronic device to: 

determine an effective rate for a virtual channel in a node 
including to determine the total number of cells to pass 
through the node during a predetermined period of 
time, and 

calculate a portion of the predetermined period of time 
during which a queue for the virtual channel of the node 
is not empty the effective rate based on a throughput 
rate when there is data in a queue corresponding to the 
virtual channel; and 

provide the determined effective rate for a virtual channel 
to a network monitor. 

32. The article of manufacture of claim 31, wherein to 
determine the portion of the predetermined period of time 
during which a queue for the virtual channel of the node is 
not empty comprises maintaining a data structure represent- 
ing the virtual channel, the data structure listing one or more 
arrival times for one or more cells of the virtual channel. 

33. The article of manufacture of claim 32, wherein the 
data structure listing one or more arrival times for one or 
more cells of the virtual channel is structured as a linked list. 
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